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Background - Single Case Designs 


■ Experimental Design 

■ Repeated measurements on the same 
experimental unit overtime 



Number of Sessions 



Background - Single Case Designs 


■ Regression techniques can be used to analyze 
SCD data in addition to visual analysis 

■ One problem 

■ Observations are nested within the same participant 

■ Violates assumption that errors are independently and 
identically distributed 

■ Errors may be serially correlated (i.e. autocorrelated) 



Background - Autocorrelation 


■ Autocorrelation (AC) is typically assessed 
using a lag-i AC coefficient 


■ correlation between the regression residuals and 
the same residuals shifted ahead by one unit of 


time 


A- 




/=l 

■ Where 

■ 6f = the residual attime t 

■ A/ = number of observations in time series 


Huitenna & McKean (1991) 



Background - Autocorrelation 


How prevalent is AC? 

■ Some debate 

Shadish and Sullivan (in 
press) 

■ Analyzed 799 SCDs 

■ Average AC = -.044, p < .01 

■ Range = -.931-786 

■ Significant Heterogeneity 

■ (Q = 4306.18, p < .001, P = 81%) 



rho 




Study Purpose 


■ To get a more accurate estimate of the AC by 
modeling the hierarchical nature of the data 

■ Cases nested within participants nested within 
studies nested within journals 

■ To find factors that explain the between- 
study variability found in observed 
autocorrelation estimates 



Methods 


■ 799 SCDs from Shadish and Sullivan (in press) 

■ SCD data extracted using UnGraph 

■ 19 moderator variables coded for each case 

■ Time between observations estimated 

■ Data fit using regression models 

■ Treatment, Trend, and Interaction terms 

■ AC estimates computed with residuals. 

■ r^ estimator 

■ AC estimates analyzed using multi-level meta-analytic models 

■ Random Effects 

■ Mixed Effects 



Methods 


■ Simple regression model 

■ y=^o+ Pit/A77e + p ^treatment + p^timeHreatment 

■ Fora subset of 352 SCDs 

■ Longest SCDs (22 observations or more) to avoid perfect fit 

■ Higher order trend and interaction terms added to regression model 

■ Take into account potential non-linearity 

■ For example: 

■ Model 2 

■ y = Po+ + P^treatment + f>^time^treatment + P^t/7r?e^ + f>^time^^treatment 



Methods 


■ Full multi-level model: 

Pijklm “ Yoo ^0/ ^ok ^o/ ^om ^ijklm 
■ Where 

■ Yoo = mean 

■ ^ ~ N(o,t^) , j = case, k = participant, I = study, m = journal 

■ £ = residual error 



Methods 


■ SAS code - 2 level model 

Proc mixed method=ml covtest data=temp; 
class ids pid sid jid; 
model rho= / ddfm=satterth s cl; 
random int / subject=ids; 
repeated /group=ids; 
parms / parmsdata=betvar 
eqcons = 2 to 800, ■ 
run; 



Methods 


■ SAS code - 5 level model 

Proc mixed method=ml covtest data=temp; 
class ids pid sid jid; 
model rho= / ddfm=satterth s cl; 
random int / subject =jid; 
random int / subject=sid(jid); 
random int/subject=pid(sidjid); 
random int/subject=ids(pid sid Jid); 
repeated /group=ids; 
parms / parmsdata=betvar 
eqcons = sto 803; 
run; 



Results 


■ Unconditional model results 



2-Level 

Model 

3 -Level 
Model 

4-Level 

Model 

5 -Level 
Model 

Fixed Effects 

Intercept 

-.028* 

-.041* 

-.036 

-.045 

Random 

Effects 

Cases 

.101*** 

.031*** 

.031*** 

.031*** 

Participants 

- 


2e-4 

2e-4 

Studies 

- 

- 

.073*** 

Q'y 

Journals 

- 

- 

- 

.004 

-2Loglikelihood 

723.0 

598.1 

415.3 

415.2 


Note: N = 799. *j!7<.05. ***j!7<.001. 



Results 


■ Conditional model 
results 


5-Level Model 

Random Effects 


Cases 

026*** 

Participants 

.002 

Studies 


Journals 

.002 


Note: Satterthwaite degrees of freedom used; rounded to 
nearest interger. 

t/7<.10. *p < .05. **;?<. 01. ***/?<.001. 


5-Level Model 

Fixed Effects 


DV ID 

F( 1,732) = 0.43 

SCD Design 

OO 

oo 

II 

CD 

DV Direction 

F(l,578)= 1.32 

DV Metric 

F(7,336)= 1.33 

Participant has Autism 

F(l,45) = 0.07 

Educational Study 

F(l,92) = 0.00 

Ceiling/Floor Effects 

F(2,751) = 0.07 

Ease of Coding Time 

F(4,86) = 0.55 

Who Coded DV 

F(3,109) = 0.57 

DV Content 

F(3,173) = 0.23 

Number of Sessions 

F(l,309) = 44.44*** 

Average Time between Sessions 

F(l,149)= 1.09 

DV Changeability 

F(l,742) = 3.0L 

Participant Age 

F(3,193) = 3.45* 

Participant is a Student 

F(l,157) = 2.62 

Participant has Develop. Disord 

F(l,172) = 10.23** 

Participant has Clin. Diagnosis 

F(l,174) = 0.14 

Location of Study 

F(6,105)= 1.47 

Acceptable Level of IRR 

F(4,429) = 2.35t 



Results 


■ Results from investigating non-linearity 


Model 

Parameters 

T 2 

^Oj 

-2Loglikelihood 

Model 1 

3 

.110 

286.8 

Model 3 

7 

.081 

196.7 

Model 10 

21 

.060 

105.9 


Note: N = 352. 



Discussion 



Estimating the AC 

■ Average AC not significantly different from zero 
after modeling the full data structure. 



Discussion 


■ Reducing heterogeneity 

■ Modeling the full data structure reduces between- 
case heterogeneity substantially (from .101 to .031) 

■ Adding moderator variables further reduces 
between-case heterogeneity (from .031 to .026) 

■ Adding moderator variables also reduces between- 
study heterogeneity (from .071 to .047) 

■ Modeling non-linearity reduces between-case 
heterogeneity substantially (from .110 to .060) 



Discussion 


■ Significant moderators 

■ Number of sessions 

■ Positive relationship with AC 

■ Short time series are negatively biased 

■ Participant age 

■ Adults associated with lowest levels of AC 

■ Teens associated with highest levels of AC 

■ Developmental disorder 

■ Those with developmental disorders associated with 
lower levels of AC 



Conclusions 


■ The nesting of meta-analytic data should be 
modeled when possible 

■ Autocorrelation may only be an issue in SCDs 
with specific characteristics 

■ A non-trivial amount of non-linearity is likely 
present in SCD data 



Limitations and Future Directions 


■ Non-linearity findings may not be 
generalizable to short time series 

■ No correction used for multiple covariate 
significance tests 

■ Additional case- and study-level moderators 
should be investigated to help explain 
remaining heterogeneity 
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